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A new method for evaluating cancer education 
programs, using an external/internal evaluation team is outlined. The 
internal program staff are requirsd to collect the data, arrange for 
a site visit, provide access to personnel, and make available other 
information requested by the evaluators. The external team consists 
of a dentist with oncological background and an educational 
evaluation specialist. The evaluation is conducted in three phases: 
previsit preparation, on-site visit, and report preparation. This 
method provi^^i^s feedback on the process as well as on program 
outcomes. It focuses on improvement of the program, rather than 
success of the project. It gives appropriate attention to all ~3tages 
of program activity. It cost- about the same to implement an internal 
evaluation, but maintains tht. integrity of an external evaluation 
approach. The model has been used successfully for four years at the 
Louisiana State University School of Dentistry Clinical Cancer 
Education Program with several different evaluators. The project 
administrators, evaluators, and the funding agency have determined 
the results to be satisfactory. (DWH) 



ED 252 579 

AUTHOR 
TITLE 

PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 

ABSTRACT 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 



THE DBVELOPHBHT AND ZHPLEMEMTAIION 
OF k MDDEL FOE EVALUAIIMG 
CLINICAL SPECULTT EDUCATION PBOGEAMS 



U.S. DIPARTMENT OF EDUCATION 

NATIONAL INSTITUTE OF EDUCATION 

I I^UCATIONAL RESOURCtS INFORMATION 
CENTER (ERICI 
^ Tfus (locurnuni hds boen reproduced as 
rtM.civttil ffoin iho (lerson or organization 

Minor chdfujos havci been made to improve 
fi'produchoi) quality 



• Poinis o\ viL»w or opinions stated in this docu 
moni do nut nocossarilv represent oHicial NIE 
position or policy 



James E. McLean 
Evaluation and Assessment Laboratory 
The University of Alabama 



Lynda Nevby Lee 
Administration 9 Counseling » and Educational Studies 
Eastern Kentucky University 

Mickey M. Lee 
Administration 9 Counseling » and Educational Studies 
Eastern Kentucky University 

"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER <ERICr 



Presented at the Thirteenth Annual Meeting 

of the 

Mld'South Educational Research Association 
New Orleans » Louisiana 
November » 1984 



ERIC 



McUan, Lee, & Lee Evaluating Clinical Specialties 

Abstract 

Although the need for evaluations and accountability has been well 
documented, good evaluations of educational programs have been rare! This has 
been particularly true in medical and dental education where many evaluations 
are based upon the "I feel or I think it was effective" model. However, the 
need for greater accountability has been brought into focus by such governing 
bodies as the National Cancer Institute and the National Institute for 
Education. As the pressure for effectively documenting program change has 
increased, the proliferation of evaluation models Has expanded. Many of these 
models, however, have limited applicability to medical/dental heath centers. 
The focus of this paper Is on the development and implementation of a model 
for evaluating clinical specialty education programs. 

The model proposed in this paper is based on an external/ internal 
approach: staff are required to collect data and make arrangements for the 
external evaluation team-ra dentist with an oncological background and an 
educational evaluation specialist. The evaluation is conducted in three 
phases: prevlsit preparation, on-site visit, and report preparation. 

The major advantage of the evaluation method described in this paper is 
that it focuses on improvement of the program rather than to indicate whether 
or not the program was successful. This model also gives appropriate 
attention to all stages of program activity. It costs about the same to 
implement an internal evaluation, but maintains the integrity of an external 

evaluation approach. 

The Louisiana State University School of Dentistry Clinical Cancer 
Educatiou Program has used this model successfully for four years with several 
different evaluators. Results have proved satisfactory from the viewpoints of 
the evaluators, the project administrators, and the funding agency. A 
discussion of the model and its wider implications is provided. 
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THB DBmOPMDR AND XMPLEMENTAIZON 
OP ▲ MODEL POR BVALUmilO 
CLINICAL SPECIALTY EDUCATION PROGRAMS 

Although the need for evaluations and accountability has been well 
documented, good evaluations of educational programs have been rare (Worthen & 
Sanders, 1973). This appears to be particularly true In inedlcal/ dental 
education where many evaluations are based upon the **I feel or I think It was 
effective*" model (Cassldy, 1983; McLean & Lee, 1982) • However, the need for 
greater accountability has been brought Into focus by such governing bodies as 
the National Cancer Institute and the National Institute of Education, among 
others. As the pressure to effectively document program changes has 
Increased, there has been a proliferation of evaluation models (Mltzel, 1982), 
but many of these models have limited applicability to medical/ dental health 
centers • 

The purpose of this paper Is to outline a new method for evaluating 
cancer education programs: using an external/ Internal evaluation team. This 
model provides feedback on the process as well as on ptogram outcomes and Is 
based on review of data from a multitude of sources. Including various program 
participants, materials produced by the program, and testing results* 

Description of the Model 

The model Is based on an Internal/ external 'approach* The Internal 
program staff are required to collect the data, arrange for a site visit, 
provide access to personnel, and make available other Information requested by 
the evaluators* The external evaluation team consists of two persons, a 
dentist with an oncological background and an educational evaluation 
specialist* 

The evaluation Is conducted In three phases«*-prevlslt preparation 
on*-slte visit, and report preparation. During the previsit, the oncologist 
and the evaluator become familiar with the .program and determine their on-^slte 
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needs. Then a prearranged schedule which includes tine for orientation, 
interviews, examination of program materials and records, inspection of the 
facilities, and an exit interview with the program director and staff is 
provided. Finally, preparation of the report includes analysis of the data 
and integration of the findings. Conclusions and recommendations are provided 
during this final stage. 
Onsite Visit Preparation; Phase One 

Preparation can begin as soon as the evaluation team has been designated. 
Previsit preparation must be done by both members and the on-site program 
director and is essential to a productive on-site visit. 

The evaluation team members must become familiar with the program and 
determine on-site needs. Program materials such as the program proposal, 
which usually defines the objectives as well as the activities designed to 
accomplish those objectives, must be Inspected. The team muse also state what 
interviews, facility inspections, and program materials they will need. This 
previsit preparation is essentially an input evaluation, through which the 
team can make initial Judgments about the adequacy of the project's resources 
for accomplishidng its objectives, while preparing themselves for the on-site 
visit. 

Previsit preparation by program administrators is primarily logistical: 
identifying the evaluators, providing them with necessary materials, arranging 
schedules for interviews and tours, and providing work space. These things 
must be done so that the site visit can be completed in about two days. 

If the team has requested appropriaate information and been provided it, 
the stage is set. The on-site visit is the most costly phase of the 
evaluation, so inadequate preparation by either the team or the program 
personnel must be avoided. 
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The On-alte Vlelt: Phase Two 

I 

The on«^8lte visit Is the keystone of the evaluation model j^escrlbed In 
this paper: the evaluators acquire the rest of the Information they need 
through Interviews, observations, and examination of outcome data, for 
completing phase three**-the report. 

The on*-slte visit should Include time for a complete briefing of the 
evaluators at the beginning and an exit Interview at the end* Questions 
raised In the study of the program can be answered and program personnel met 
at the beginning. The exit Interview provides further opportunity to clarify 
and to follow up on Information gathered during the visit, and allows feedback 
of Initial Impressions to program administrators. 

The evaluators conduct Interviews, tour facilities, review materials, 
then discuss their Information between orientation and the exit Interview. 
The visit can be usually completed In a two-day period, assuming that the 
evaluators and the program personnel are both prepared beforehand. 

Interviews • Interviews provide a direct source of Information, but the 
right questions must be asked of the right people, who may Include anyone 
directly or Indirectly connected with the program. 

The choice of whom to Interview Is usually a deliberate two-step 
procedure because of the time limits* First, the type must be determined: 
whether faculty members, administrators, students, staff, or patients, etc> » 
and availability will certainly play a role In the selection process. Second, 
specific Individuals must be selected. Relevance Is the first criterion, so 
those who have the most knowledge of the program should be selected. If the 
number Is large, a random sample should be used. 

Observations . The two primary targets are facilities and materials. 
Each must contribute to meeting objectives. 

« 4 - 
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The facilities must be adequate. Classroom tests, and records, and other 
curricular materials can be Inspected. Their evaluatiodn is also based on the 
program objectives and other relevant criteria. Tests can be evaluated on the 
basis of validity and reliability. Curricular materials can be Judged on 
accuracy, clarity, and appeal, as well as their relationship to objectives. 

Outcome data t Although program outcome is an important component of a 
complete evaluation, it is of much less value without interviews and 
observations. A traditional outcome evaluation would use a control group to 
obtain an estimate of what the accomplilshment of the target participants 
would have been without benefit of the program. Since control groups are not 
usually feasible in cancer education programs, the estimate has to be obtained 
in one of three other ways. 

The first is to use a normative approach. This requires the use of a 
test or instrument for which normative data are available on a similar 
population and that uses the normative populatiodn as a control group. 

The second is collection of data over several years, often referred to as 
a Time Series Design, in which the program treatment group acts as its own' 
control. The major advantage of this approach is that it does not require the 
use of a previously normed instnunent; the major disadvantage is that it 
requires data to be collected before the program is implemented. 

The third is the use of archival data collected as part of the normal 
program, including end**of*year or graduation tests, attendance records, 
clinical logs, etc. The major disadvantage of using this type of data is that 
its availability is often a matter of luck. 

Outcome evaluation should also be guided by the program's objectives and 
would usually include attitudinal and performance (clinical) outcomes as well 
as cognitive outcomes. Appropriate measures of these is highly important and 
Is dependent on skillful design. 
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Report Preparation! Phase Three 

Preparation of the evaluator's report also requires three phases— sharing . 
information, analyzing the data, and preparing the report. Most of the work 
is done subsequent to the visit and involves both members. The evaluation 
specialist will take the leadership role at this stage by merging the 
information, analyzing the data, and preparing the draft of the report. The 
oncologist will participate by sharing the information collected during the 
visit, including personal observations, and critiqueing the draft of the 
evaluation report. 

At this point, it is easy to see why the team concept is so important to 
the evaluation. Though the oncologist and the evaluation specialist may have 
observed the same things during their on-site visit, they often comprehend and 
interpret them in different ways. For example, the oncologist may be Judging 
the content validity of a classroom test while the evaluation specialist will 
be judging its reliability. The different perspectives are what gives depth 
to the evaluation. 

An evaluator can analyze the information in a number of ways; however, 
much of it can be analyzed using content analysis (Lee & McLean, 1978). Some 
of the data will also be amenable to more traditional quantitative analyses. 
For the most part, the data analyses serve to summarize the data for the 
evaluation report. 

The evaluation report provides feedback to the program, so it should be 
complete, yet as concise as possible. There are a ntunber of acceptable 
methods for organizing the evaluation report (McLean & Lee, 1982). The 
overriding criterion should be the report's precision of analysis, not only of 
the overall effectiveness of the program, but also of its specific strengths 
and weaknesses, followed by recommendations. 
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Discussion 

The major advantage of the evaluation method described In this paper Is 
that It takes Into account the current developmental stage of the program, 
l*e> the purpose of the evaluation model Is not merely to Indicate whether or 
not the program was successful, but to Indicate how It can be Improved. Thus, 
several aspects of the model should not be Ignored # These aspects are the 
evaluation team concept, the multlyear approach, and the early emphasis on 
process rather than on outcome # 

The evaluation team must Include both an oncologist and an evaluation 
specialist # The evaluation specialist will design and Implement the 
evaluation, but needs ^ the advice of the oncologist on content and 
Interpretation. Similarly, the oncologist will examine and judge the content, 
but probably will not be able to completely analyze the results. 

It Is important that the evaluation be repeated periodically. A new 
program generally has three distinct phases^-'developmental, testing, and 
validation. Each calls for a different emphasis In the evaluation. In the 
developmental phaSe, emphasis should be on the process and be formative. 
During the test phase, where collection of outcome data has begun, process 
i>valuatlon Is still emphasized, but the outcome evaluation Is also necessary. 
The validation phase calls for a complete outcome evaluatlodn to determine 
whether or not the program Is effective. 

Evaluation Is often Interpreted to mean only outcome evaluation. Process 
evaluation Is often Ignored even' though It Is particularly Important in the 
developmental and testing phases of a program. The model presented In this 
paper gives appropriate attention to all stages of program activity. Failure 
in Implementation often results In effective programs; this model of 
evaluation also provides for proper attention to program administration. 
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The evaluation approach described In this paper is also economical. A 
two-day visit, lO-person-days of outside assistance, four days from an 
oncologist and six from an evaluation specialist are required. 

Summary 

The evaluation model described In this paper Is a comprehensive 
evaluation approach using an external evaluation team, an oncologist and a 
program evaluation specialist, who cooperate in preparing and conducting an 
onslte visit, and writing an evaluation report. The mod<dl costs about the 
same to implement as an internal evaluation, but maintains the integrity of an 
external evaluation. 

The model is now in its fourth year of use by the Louisiana State 
University School of Dentistry Clinical Cancer Education Program. Results 
from the first three years have proved satisfactory from the viewpoints of the 
evaluators, the project administrators, and the funding agency ."^ Second, 
third, and fourth year evaluators found that numerous first year problems were 
solved as a result of an evaluation according to this method. 



ERIC 



McLean, Ue, & Lee Evaluating Clinical Specialties 

References 

Casaidy R. (1983). General practice residency and advanced education in 
general dentistry programs! Evaluation. Journal of Dental Education, 
47_(6), 398-405. 

Evaluation of programs (1982). H. Mitzel (Ed.), Encyclopedia of educational 
research ., (5th ed.) (pp. 592-611). New York: The Free Press. 

McLean, J., & Lee, M. (1982). A model for external evaluation cancer 
educational programs . Paper presented at the meeting of the American 
Association for Cancer Education, Birmingham, Alabama. 

Worthen, B., & Sanders. J. (1973). Educational evaluation: Theory and 
practice . Belmont, CA: Wadsworth. 



